---
title: "OpenAI SDK Integration"
description: "Understanding MCPJam Inspector's OpenAI Apps SDK implementation for custom UX components"
icon: "puzzle-piece"
---

# OpenAI SDK Architecture

This guide explains how MCPJam Inspector implements the OpenAI Apps SDK to render custom UI components for MCP tool results. This enables MCP server developers to create rich, interactive visualizations for their tool outputs.

<Note>
  MCPJam Inspector also supports
  [MCP-UI](https://github.com/modelcontextprotocol/mcp-ui) for simpler
  component-based UIs. See the [Playground
  Architecture](./playground-architecture#mcp-ui-integration) docs for MCP-UI
  implementation details.
</Note>

## Overview

MCPJam Inspector provides full support for the [OpenAI Apps SDK](https://developers.openai.com/apps-sdk), allowing MCP tools to return custom UI components that render in iframes with a sandboxed `window.openai` API bridge.

<Note>
  This document covers the **V1 Playground** implementation. As of PR #773,
  **Playground V2** (`ChatTabV2.tsx`) also supports OpenAI Apps with a
  streamlined implementation using `openai-app-renderer.tsx`. The V2
  implementation uses MCP resources API to fetch widget templates and renders
  them with similar `window.openai` bridge capabilities.
</Note>

### Key Features

- **Custom UI Rendering**: Display tool results using custom HTML/React components
- **Interactive Widgets**: Components can call other MCP tools and send followup messages
- **State Persistence**: Widget state persists across sessions via localStorage
- **Theme Synchronization**: Widgets automatically receive theme updates (light/dark mode)
- **Secure Isolation**: Components run in sandboxed iframes with CSP headers
- **Server-Side Storage**: Widget context stored server-side with 1-hour TTL for iframe access
- **Modal Support**: Widgets can open modal dialogs with separate view contexts
- **Dual Mode Support**:
  - `ui://` URIs for server-provided HTML content
  - External URLs for remotely hosted components

### OpenAI Apps SDK vs MCP-UI

MCPJam Inspector supports two approaches for custom UI rendering:

| Feature           | OpenAI Apps SDK                   | MCP-UI                         |
| ----------------- | --------------------------------- | ------------------------------ |
| Specification     | OpenAI proprietary                | MCP-UI (open standard)         |
| Rendering         | Sandboxed iframes                 | RemoteDOM components           |
| Complexity        | Full web applications             | Simple interactive components  |
| Tool calls        | `window.openai.callTool()`        | Action handlers                |
| State persistence | `window.openai.setWidgetState()`  | Not supported                  |
| Security          | Full iframe sandbox               | Component-level isolation      |
| Best for          | Complex dashboards, charts, forms | Buttons, cards, simple layouts |

**When to use OpenAI Apps SDK:**

- Need full JavaScript framework support (React, Vue, etc.)
- Require persistent state across sessions
- Building complex interactive visualizations
- Need access to external APIs and libraries

**When to use MCP-UI:**

- Simple interactive components (buttons, cards)
- Prefer open standards over proprietary APIs
- Don't need state persistence
- Want faster rendering without iframe overhead

See [Playground Architecture - MCP-UI Integration](./playground-architecture#mcp-ui-integration) for MCP-UI implementation details.

## Architecture Overview

```mermaid
graph TB
    subgraph Frontend["Frontend (React)"]
        Chat[ChatTab Component]
        Message[Message Component]
        Results[ResultsPanel Component]
        Renderer[OpenAIComponentRenderer]
        Iframe[Sandboxed Iframe]

        Chat --> Message
        Message --> Results
        Results --> Renderer
        Renderer --> Iframe
    end

    subgraph Backend["Backend (Hono.js)"]
        ToolExec[Tool Execution]
        Storage[Widget Data Store]
        StoreEndpoint["POST /openai/widget/store"]
        WidgetEndpoint["GET /openai/widget/:toolId"]
        ContentEndpoint["GET /openai/widget-content/:toolId"]
        MCPRead[MCP Resource Read]

        ToolExec --> StoreEndpoint
        StoreEndpoint --> Storage
        WidgetEndpoint --> ContentEndpoint
        ContentEndpoint --> Storage
        ContentEndpoint --> MCPRead
    end

    subgraph MCPServer["MCP Server"]
        Tool[MCP Tool]
        Resource["Resource: ui://"]

        Tool -.returns.-> Resource
    end

    Renderer -->|Store widget data| StoreEndpoint
    Renderer -->|Load iframe| WidgetEndpoint
    Iframe -->|postMessage| Renderer
    Renderer -->|Call tool| ToolExec
    ContentEndpoint -->|Fetch HTML| MCPRead
    MCPRead -->|Read resource| Resource

    classDef frontend fill:#e1f5ff,stroke:#0288d1
    classDef backend fill:#fff3e0,stroke:#f57c00
    classDef mcp fill:#f3e5f5,stroke:#7b1fa2

    class Chat,Message,Results,Renderer,Iframe frontend
    class ToolExec,Storage,WidgetEndpoint,ContentEndpoint,MCPRead backend
    class Tool,Resource mcp
```

## Component Flow

### 1. Tool Execution & Detection

When a tool is executed that returns OpenAI SDK components or MCP Apps, the system detects this in multiple ways:

**Method A: `_meta["openai/outputTemplate"]` field (ChatGPT Apps)**

```typescript
// Tool definition includes OpenAI output template
{
  "name": "weather_widget",
  "description": "Get weather with custom UI",
  "_meta": {
    "openai/outputTemplate": "ui://weather/display"
  }
}
```

**Method B: `_meta["ui/resourceUri"]` field (MCP Apps)**

```typescript
// Tool definition includes MCP Apps resource URI
{
  "name": "weather_widget",
  "description": "Get weather with custom UI",
  "_meta": {
    "ui/resourceUri": "ui://weather/display"
  }
}
```

**Method C: `ui://` resource in response**

```typescript
// Tool result contains ui:// resource
{
  "content": [{
    "type": "resource",
    "resource": {
      "uri": "ui://weather/display",
      "mimeType": "text/html"
    }
  }]
}
```

#### ResultsPanel Detection Logic

Located in `client/src/components/tools/ResultsPanel.tsx:100-104`:

```typescript
// Check for OpenAI component or MCP App using tool metadata from definition
const openaiOutputTemplate = toolMeta?.[\"openai/outputTemplate\"];
const mcpAppResourceUri = toolMeta?.[\"ui/resourceUri\"];
const hasOpenAIComponent =
  openaiOutputTemplate && typeof openaiOutputTemplate === \"string\";
const hasMCPApp = mcpAppResourceUri && typeof mcpAppResourceUri === \"string\";
const uiResource = resolveUIResource(result);
```

The system detects custom UI components by checking for:

1. `openai/outputTemplate` metadata (ChatGPT Apps)
2. `ui/resourceUri` metadata (MCP Apps)
3. `ui://` URIs in tool results via `resolveUIResource`

The `resolveUIResource` function searches for `ui://` URIs in:

1. Direct `resource` field at root level
2. `content` array items with `type: "resource"`

### 2. Widget Data Storage Flow

Before rendering, widget data must be stored server-side for iframe access:

```mermaid
sequenceDiagram
    participant Renderer as OpenAIComponentRenderer
    participant API as API Endpoint
    participant Store as Widget Data Store

    Renderer->>Renderer: Extract structuredContent from result
    Renderer->>API: POST /api/mcp/openai/widget/store
    API->>Store: Store widget data with toolId
    Store-->>API: Success
    API-->>Renderer: Return success: true
    Renderer->>Renderer: Set widget URL
    Note over Store: Data expires after 1 hour TTL
```

**Why Store Server-Side?**

- Iframes need access to `toolInput` and `toolOutput` for `window.openai` API
- Client localStorage can't be shared across iframe sandbox boundaries
- Server becomes the source of truth for widget initialization data

#### Storage Implementation

Located in `server/routes/mcp/openai.ts:14-32`:

```typescript
interface WidgetData {
  serverId: string;
  uri: string;
  toolInput: Record<string, any>;
  toolOutput: any;
  toolResponseMetadata?: Record<string, any> | null;
  toolId: string;
  toolName: string;
  theme?: "light" | "dark";
  timestamp: number;
}

const widgetDataStore = new Map<string, WidgetData>();

// Cleanup expired widget data every 5 minutes
setInterval(
  () => {
    const now = Date.now();
    const ONE_HOUR = 60 * 60 * 1000;
    for (const [toolId, data] of widgetDataStore.entries()) {
      if (now - data.timestamp > ONE_HOUR) {
        widgetDataStore.delete(toolId);
      }
    }
  },
  5 * 60 * 1000,
).unref();
```

### 3. Two-Stage Widget Loading

The system uses a clever two-stage loading process to ensure React Router compatibility:

```mermaid
sequenceDiagram
    participant Browser
    participant Stage1 as Stage 1 Container
    participant Stage2 as Stage 2 Content
    participant MCP as MCP Server

    Browser->>Stage1: GET /widget/abc123
    Stage1-->>Browser: Return container HTML
    Note over Browser: Execute history.replaceState to change URL to /
    Browser->>Stage2: Fetch /widget-content/abc123
    Stage2->>MCP: Read resource from server
    MCP-->>Stage2: Return HTML content
    Stage2->>Stage2: Inject window.openai API script
    Stage2->>Stage2: Add CSP headers
    Stage2-->>Browser: Return enhanced HTML
    Browser->>Browser: Render widget via document.write
```

**Why Two Stages?**

- Many widgets use React Router's `BrowserRouter` which expects clean URLs
- Stage 1 changes URL to "/" before widget code loads
- Stage 2 fetches actual content after URL is reset
- Prevents routing conflicts and 404 errors

#### Stage 1: Container Page

Located in `server/routes/mcp/openai.ts:81-120`:

```typescript
openai.get(\"/widget/:toolId\", async (c) => {
  const toolId = c.req.param(\"toolId\");
  const widgetData = widgetDataStore.get(toolId);

  if (!widgetData) {
    return c.html(\"Error: Widget data not found or expired\", 404);
  }

  return c.html(`
    <!DOCTYPE html>
    <html>
    <head>
      <meta charset=\"utf-8\">
      <title>Loading Widget...</title>
    </head>
    <body>
      <script>
        (async function() {
          // Change URL to \"/\" BEFORE loading widget
          history.replaceState(null, '', '/');

          // Fetch actual widget HTML
          const response = await fetch('/api/mcp/openai/widget-content/${toolId}');
          const html = await response.text();

          // Replace entire document
          document.open();
          document.write(html);
          document.close();
        })();
      </script>
    </body>
    </html>
  `);
});
```

#### Stage 2: Content Injection

Located in `server/routes/mcp/openai.ts:123-459`:

Key steps:

1. Retrieve widget data from store
2. Read HTML from MCP server via `readResource(uri)`
3. Parse view mode and params from query string (`view_mode`, `view_params`)
4. Inject `window.openai` API script with bridge implementation
5. Add security headers (CSP, X-Frame-Options)
6. Set cache control headers (no-cache for fresh content)

### 4. window.openai API Bridge

The injected script provides the OpenAI Apps SDK API to widget code:

```mermaid
graph LR
    subgraph IframeWidget["Iframe (Widget)"]
        Widget[Widget Code]
        API[window.openai]
    end

    subgraph ParentInspector["Parent (Inspector)"]
        Renderer[OpenAIComponentRenderer]
        Backend[Backend API]
    end

    Widget -->|callTool| API
    API -->|postMessage| Renderer
    Renderer -->|fetch tools API| Backend
    Backend -->|postMessage response| Renderer
    Renderer -->|postMessage result| API
    API -->|Promise resolves| Widget

    Widget -->|setWidgetState| API
    API -->|localStorage and postMessage| Renderer

    Widget -->|sendFollowupTurn| API
    API -->|postMessage| Renderer
    Renderer -->|Sends to chat| Backend

    classDef iframe fill:#ffe0b2,stroke:#e65100
    classDef parent fill:#e1f5ff,stroke:#0288d1

    class Widget,API iframe
    class Renderer,Backend parent
```

#### API Implementation

Located in `server/routes/mcp/openai.ts:213-376`:

**Core API Methods:**

```javascript
const openaiAPI = {
  toolInput: ${JSON.stringify(toolInput)},
  toolOutput: ${JSON.stringify(toolOutput)},
  toolResponseMetadata: ${JSON.stringify(toolResponseMetadata ?? null)},
  displayMode: 'inline',
  maxHeight: 600,
  theme: ${JSON.stringify(theme ?? "dark")},
  locale: 'en-US',
  safeArea: { insets: { top: 0, bottom: 0, left: 0, right: 0 } },
  userAgent: {
    device: { type: 'desktop' },
    capabilities: { hover: true, touch: false }
  },
  view: {
    mode: ${JSON.stringify(viewMode)},  // 'inline' or 'modal'
    params: ${serializeForInlineScript(viewParams)}
  },
  widgetState: null,

  // Persist widget state (localStorage + propagate to model)
  async setWidgetState(state) {
    this.widgetState = state;
    try {
      localStorage.setItem(widgetStateKey, JSON.stringify(state));
    } catch (err) {
      console.error('[OpenAI Widget] Failed to save widget state:', err);
    }
    // Propagate state to parent for model context
    window.parent.postMessage({
      type: 'openai:setWidgetState',
      toolId: toolId,
      state
    }, '*');
  },

  // Call another MCP tool
  async callTool(toolName, params = {}) {
    return new Promise((resolve, reject) => {
      const requestId = `tool_${Date.now()}_${Math.random()}`;

      const handler = (event) => {
        if (event.data.type === 'openai:callTool:response' &&
            event.data.requestId === requestId) {
          window.removeEventListener('message', handler);
          event.data.error ? reject(new Error(event.data.error))
                           : resolve(event.data.result);
        }
      };

      window.addEventListener('message', handler);
      window.parent.postMessage({
        type: 'openai:callTool',
        requestId,
        toolName,
        params
      }, '*');

      setTimeout(() => {
        window.removeEventListener('message', handler);
        reject(new Error('Tool call timeout'));
      }, 30000);
    });
  },

  // Send followup message to chat
  async sendFollowupTurn(message) {
    const payload = typeof message === 'string'
      ? { prompt: message }
      : message;
    window.parent.postMessage({
      type: 'openai:sendFollowup',
      message: payload.prompt || payload
    }, '*');
  },

  // Request display mode change
  async requestDisplayMode(options = {}) {
    const mode = options.mode || 'inline';
    const maxHeight = options.maxHeight;
    this.displayMode = mode;
    if (typeof maxHeight === 'number') {
      this.maxHeight = maxHeight;
    }
    window.parent.postMessage({
      type: 'openai:requestDisplayMode',
      mode,
      maxHeight
    }, '*');
    return { mode };
  },

  // Alias for compatibility
  async sendFollowUpMessage(args) {
    const prompt = typeof args === 'string' ? args : (args?.prompt || '');
    return this.sendFollowupTurn(prompt);
  },

  // Open external URL
  async openExternal(options) {
    const href = typeof options === 'string' ? options : options?.href;
    if (!href) {
      throw new Error('href is required for openExternal');
    }
    window.parent.postMessage({
      type: 'openai:openExternal',
      href
    }, '*');
    window.open(href, '_blank', 'noopener,noreferrer');
  },

  // Request modal dialog
  async requestModal(options) {
    window.parent.postMessage({
      type: 'openai:requestModal',
      title: options.title,
      params: options.params,
      anchor: options.anchor
    }, '*');
  }
};

// Make available globally
window.openai = openaiAPI;
window.webplus = openaiAPI; // Compatibility alias
```

**Security Notes:**

- API is frozen with `writable: false, configurable: false`
- 30-second timeout on tool calls prevents hanging requests
- Origin validation in parent ensures only iframe messages are processed

### 5. Modal Support

As of PR #931, widgets can request modal dialogs using `window.openai.requestModal()`. This enables widgets to display secondary views or detailed information in a separate modal context.

#### Modal Architecture

```mermaid
graph TB
    subgraph InlineWidget[Inline Widget Iframe]
        InlineCode[Widget Code]
        InlineAPI[window.openai]
    end

    subgraph ModalWidget[Modal Widget Iframe]
        ModalCode[Widget Code]
        ModalAPI[window.openai]
    end

    subgraph Parent[OpenAIComponentRenderer]
        Dialog[Dialog Component]
        StateSync[State Synchronization]
    end

    InlineCode -->|requestModal| InlineAPI
    InlineAPI -->|postMessage| Parent
    Parent -->|Opens| Dialog
    Dialog -->|Mounts| ModalWidget
    ModalCode -->|Receives view.mode='modal'| ModalAPI

    InlineAPI -->|setWidgetState| StateSync
    StateSync -->|pushWidgetState| ModalAPI
    ModalAPI -->|setWidgetState| StateSync
    StateSync -->|pushWidgetState| InlineAPI

    classDef inline fill:#e1f5ff,stroke:#0288d1
    classDef modal fill:#fff3e0,stroke:#f57c00
    classDef parent fill:#f3e5f5,stroke:#7b1fa2

    class InlineCode,InlineAPI inline
    class ModalCode,ModalAPI modal
    class Dialog,StateSync parent
```

#### Modal Request Flow

Located in `client/src/components/chat-v2/openai-app-renderer.tsx:380-388`:

```typescript
case "openai:requestModal": {
  setModalTitle(event.data.title || "Modal");
  setModalParams(event.data.params || {});
  setModalOpen(true);
  break;
}
```

When a widget calls `window.openai.requestModal()`, the parent:

1. Extracts modal title and params from the message
2. Opens a Dialog component
3. Mounts a new iframe with the same widget URL
4. Appends query params: `?view_mode=modal&view_params=<encoded_json>`

#### View Mode Detection

Widgets receive view context via `window.openai.view`:

```javascript
// In widget code
if (window.openai.view.mode === "modal") {
  // Render modal-specific UI
  const params = window.openai.view.params;
  renderModalView(params);
} else {
  // Render inline UI
  renderInlineView();
}
```

#### Widget State Synchronization

Modal and inline views share widget state automatically. When either view calls `setWidgetState()`, the state is propagated to the other view via `openai:pushWidgetState` messages.

Located in `client/src/components/chat-v2/openai-app-renderer.tsx:269-284`:

```typescript
// After state is set, push to other view
const targetWindow = isFromInline
  ? modalWindow
  : isFromModal
    ? inlineWindow
    : null;

if (targetWindow) {
  targetWindow.postMessage(
    {
      type: "openai:pushWidgetState",
      toolId: resolvedToolCallId,
      state: event.data.state,
    },
    "*",
  );
}
```

The receiving view updates its local state and dispatches a `openai:widget_state` event:

Located in `server/routes/mcp/openai.ts:410-428`:

```javascript
if (
  event.data.type === "openai:pushWidgetState" &&
  event.data.toolId === toolId
) {
  try {
    const nextState = event.data.state ?? null;
    window.openai.widgetState = nextState;

    // Update localStorage
    try {
      localStorage.setItem(widgetStateKey, JSON.stringify(nextState));
    } catch (err) {}

    // Dispatch event for widget to listen
    try {
      const stateEvent = new CustomEvent("openai:widget_state", {
        detail: { state: nextState },
      });
      window.dispatchEvent(stateEvent);
    } catch (err) {
      console.error(
        "[OpenAI Widget] Failed to dispatch widget state event:",
        err,
      );
    }
  } catch (err) {
    console.error("[OpenAI Widget] Failed to apply pushed widget state:", err);
  }
}
```

**Example Usage:**

```javascript
// In inline widget
document.getElementById("details-btn").addEventListener("click", () => {
  window.openai.requestModal({
    title: "Detailed View",
    params: { itemId: 123, view: "details" },
  });
});

// Widget code handles both views
if (window.openai.view.mode === "modal") {
  const { itemId, view } = window.openai.view.params;
  renderDetailedView(itemId, view);
} else {
  renderSummaryView();
}

// State changes sync automatically
window.openai.setWidgetState({ selectedItem: 123 });
// Both inline and modal views receive the update
```

### 6. Display Mode Support

As of PR #927, widgets can request different display modes to optimize their presentation:

- **Inline** (default) - Widget renders within the chat message flow with configurable height
- **Picture-in-Picture (PiP)** - Widget floats at the top of the screen in a fixed overlay
- **Fullscreen** - Widget expands to fill the entire viewport

#### Display Mode Implementation

The display mode system uses React state to track which widget (if any) is in PiP mode, and applies different CSS classes based on the current mode:

**State Management** (`client/src/components/chat-v2/thread.tsx:62-72`):

```typescript
const [pipWidgetId, setPipWidgetId] = useState<string | null>(null);

const handleRequestPip = (toolCallId: string) => {
  setPipWidgetId(toolCallId);
};

const handleExitPip = (toolCallId: string) => {
  if (pipWidgetId === toolCallId) {
    setPipWidgetId(null);
  }
};
```

**Mode Detection** (`client/src/components/chat-v2/openai-app-renderer.tsx:440-442`):

```typescript
const isPip = displayMode === "pip" && pipWidgetId === resolvedToolCallId;
const isFullscreen = displayMode === "fullscreen";
```

**CSS Classes** (`client/src/components/chat-v2/openai-app-renderer.tsx:444-476`):

```typescript
let containerClassName = "mt-3 space-y-2 relative group";

if (isFullscreen) {
  containerClassName = [
    "fixed",
    "inset-0",
    "z-50",
    "w-full",
    "h-full",
    "bg-background",
    "flex",
    "flex-col",
  ].join(" ");
} else if (isPip) {
  containerClassName = [
    "fixed",
    "top-4",
    "inset-x-0",
    "z-40",
    "w-full",
    "max-w-4xl",
    "mx-auto",
    "space-y-2",
    "bg-background/95",
    "backdrop-blur",
    "supports-[backdrop-filter]:bg-background/80",
    "shadow-xl",
    "border",
    "border-border/60",
    "rounded-xl",
    "p-3",
  ].join(" ");
}
```

**Exit Button** (`client/src/components/chat-v2/openai-app-renderer.tsx:481-493`):

```typescript
{shouldShowExitButton && (
  <button
    onClick={() => {
      setDisplayMode("inline");
      onExitPip?.(resolvedToolCallId);
    }}
    className="absolute left-2 top-2 z-10 flex h-6 w-6 items-center justify-center rounded-md bg-background/80 hover:bg-background border border-border/50 text-muted-foreground hover:text-foreground transition-colors cursor-pointer"
    aria-label="Close PiP mode"
    title="Close PiP mode"
  >
    <X className="w-4 h-4" />
  </button>
)}
```

**Key Features:**

- **Single PiP Widget**: Only one widget can be in PiP mode at a time. Requesting PiP on a different widget automatically exits the current PiP widget.
- **Automatic Inline Fallback**: If a widget is in PiP mode but another widget becomes the active PiP, the first widget automatically returns to inline mode.
- **Z-Index Layering**: Fullscreen widgets use `z-50`, PiP widgets use `z-40`, ensuring proper stacking order.
- **Transform Isolation**: The chat container uses `transform: translateZ(0)` to create a new stacking context, preventing z-index conflicts.
- **Backdrop Blur**: PiP widgets use backdrop blur for a modern floating effect with semi-transparent background.

#### Requesting Display Mode from Widgets

Widgets can request display mode changes using the `window.openai.requestDisplayMode()` API:

```javascript
// Request Picture-in-Picture mode
await window.openai.requestDisplayMode({ mode: "pip" });

// Request Fullscreen mode
await window.openai.requestDisplayMode({ mode: "fullscreen" });

// Return to inline mode
await window.openai.requestDisplayMode({ mode: "inline" });

// Set inline mode with custom height
await window.openai.requestDisplayMode({
  mode: "inline",
  maxHeight: 800,
});
```

The parent component handles these requests and updates the widget's display mode accordingly.

### 7. Parent-Side Message Handling

Located in `client/src/components/chat-v2/openai-app-renderer.tsx:312-347`:

```typescript
useEffect(() => {
  const handleMessage = async (event: MessageEvent) => {
    // Accept messages from inline or modal iframe
    const inlineWindow = iframeRef.current?.contentWindow;
    const modalWindow = modalIframeRef.current?.contentWindow;
    const isFromInline = inlineWindow != null && event.source === inlineWindow;
    const isFromModal = modalWindow != null && event.source === modalWindow;

    if (!isFromInline && !isFromModal) {
      return;
    }

    switch (event.data.type) {
      case \"openai:setWidgetState\":
        localStorage.setItem(widgetStateKey, JSON.stringify(event.data.state));
        break;

      case \"openai:callTool\":
        if (onCallTool) {
          try {
            const result = await onCallTool(
              event.data.toolName,
              event.data.params || {}
            );
            const targetWindow = event.source as Window | null;
            targetWindow?.postMessage({
              type: \"openai:callTool:response\",
              requestId: event.data.requestId,
              result: result
            }, \"*\");
          } catch (err) {
            const targetWindow = event.source as Window | null;
            targetWindow?.postMessage({
              type: \"openai:callTool:response\",
              requestId: event.data.requestId,
              error: err instanceof Error ? err.message : \"Unknown error\"
            }, \"*\");
          }
        }
        break;

      case \"openai:sendFollowup\":
        if (onSendFollowup) {
          onSendFollowup(event.data.message);
        }
        break;

      case \"openai:requestDisplayMode\":
        const mode = event.data.mode;
        setDisplayMode(mode);
        if (mode === \"pip\") {
          onRequestPip?.(resolvedToolCallId);
        } else if (mode === \"inline\" || mode === \"fullscreen\") {
          if (pipWidgetId === resolvedToolCallId) {
            onExitPip?.(resolvedToolCallId);
          }
        }
        if (typeof event.data.maxHeight === \"number\") {
          setMaxHeight(event.data.maxHeight);
        }
        break;
    }
  };

  window.addEventListener(\"message\", handleMessage);
  return () => window.removeEventListener(\"message\", handleMessage);
}, [widgetUrl, onCallTool, onSendFollowup, pipWidgetId, onRequestPip, onExitPip, resolvedToolCallId]);
```

#### Display Mode Synchronization

The component automatically resets to inline mode if another widget takes over PiP mode:

Located in `client/src/components/chat-v2/openai-app-renderer.tsx:368-372`:

```typescript
useEffect(() => {
  if (displayMode === "pip" && pipWidgetId !== resolvedToolCallId) {
    setDisplayMode("inline");
  }
}, [displayMode, pipWidgetId, resolvedToolCallId]);
```

This ensures only one widget can be in PiP mode at a time, preventing overlapping floating widgets.

#### Theme Synchronization

The parent component automatically sends theme updates to widgets when the user changes between light and dark mode:

Located in `client/src/components/chat-v2/openai-app-renderer.tsx:293-307`:

```typescript
// Send theme updates to iframe when theme changes
useEffect(() => {
  if (!isReady || !iframeRef.current?.contentWindow) return;

  iframeRef.current.contentWindow.postMessage(
    {
      type: "openai:set_globals",
      globals: {
        theme: themeMode,
      },
    },
    "*",
  );
}, [themeMode, isReady]);
```

Widgets can listen for theme changes using the `openai:set_globals` event:

```javascript
// In widget code
window.addEventListener("message", (event) => {
  if (event.data.type === "openai:set_globals") {
    const { theme } = event.data.globals;
    // Update widget UI based on theme
    document.body.classList.toggle("dark", theme === "dark");
  }
});
```

#### Widget State Propagation to Model

As of PR #891, widget state changes are now propagated to the LLM model as hidden assistant messages. This allows the AI to understand and reason about widget interactions.

**Implementation** (`client/src/components/chat-v2/openai-app-renderer.tsx:232-244`):

```typescript
case "openai:setWidgetState": {
  // Widget state is already persisted by the iframe script
  console.log("[OpenAI App] Widget state updated:", event.data.state);

  if (onWidgetStateChange && event.data.toolId === resolvedToolCallId) {
    const newState = event.data.state;
    const newStateStr = newState === null ? null : JSON.stringify(newState);

    // Dedupe: only propagate if state actually changed
    if (newStateStr !== previousWidgetStateRef.current) {
      previousWidgetStateRef.current = newStateStr;
      onWidgetStateChange(resolvedToolCallId, newState);
    }
  }
  break;
}
```

**Chat Integration** (`client/src/components/ChatTabV2.tsx:281-326`):

```typescript
const handleWidgetStateChange = useCallback(
  (toolCallId: string, state: any) => {
    setMessages((prevMessages) => {
      const messageId = `widget-state-${toolCallId}`;

      // If state is null, remove the widget state message
      if (state === null) {
        return prevMessages.filter((msg) => msg.id !== messageId);
      }

      const stateText = `The state of widget ${toolCallId} is: ${JSON.stringify(state)}`;

      const existingIndex = prevMessages.findIndex(
        (msg) => msg.id === messageId,
      );

      if (existingIndex !== -1) {
        // Update existing state message if changed
        const existingMessage = prevMessages[existingIndex];
        const existingText =
          existingMessage.parts?.[0]?.type === "text"
            ? (existingMessage.parts[0] as any).text
            : null;
        if (existingText === stateText) {
          return prevMessages; // No change
        }

        const newMessages = [...prevMessages];
        newMessages[existingIndex] = {
          id: messageId,
          role: "assistant",
          parts: [{ type: "text", text: stateText }],
        };
        return newMessages;
      }

      // Add new state message
      return [
        ...prevMessages,
        {
          id: messageId,
          role: "assistant",
          parts: [{ type: "text", text: stateText }],
        },
      ];
    });
  },
  [setMessages],
);
```

**UI Hiding** (`client/src/components/chat-v2/thread.tsx:99`):

```typescript
// Hide widget-state messages from UI (they're sent to model but not displayed)
if (message.id?.startsWith("widget-state-")) return null;
```

**Key Features:**

- **Hidden Messages**: Widget state messages are prefixed with `widget-state-` and hidden from the UI
- **Deduplication**: State changes are only propagated if the serialized state actually changed
- **Model Context**: The LLM receives state updates as assistant messages, enabling it to reason about widget interactions
- **Null Handling**: Setting state to `null` removes the state message entirely
- **Example**: When a chart widget updates its selected date range, the model receives: `"The state of widget tool_123 is: {\"startDate\":\"2024-01-01\",\"endDate\":\"2024-01-31\"}"`

This enables powerful use cases like:

- LLM understanding user interactions with widgets
- Contextual follow-up questions based on widget state
- Multi-turn conversations that reference widget selections
- Debugging widget behavior through model awareness

#### Tool Execution Bridge

Located in `client/src/components/ChatTab.tsx:181-207`:

```typescript
const handleCallTool = async (
  toolName: string,
  params: Record<string, any>
) => {
  const response = await fetch(\"/api/mcp/tools/execute\", {
    method: \"POST\",
    headers: { \"Content-Type\": \"application/json\" },
    body: JSON.stringify({
      toolName,
      parameters: params,
      // Pass serverId if only one server is connected
      ...(selectedConnectedNames.length === 1
        ? { serverId: selectedConnectedNames[0] }
        : {})
    })
  });

  const data = await response.json();
  return data.result;
};
```

## Security Architecture

### Content Security Policy

Located in `server/routes/mcp/openai.ts:408-437`:

```typescript
const trustedCdns = [
  \"https://persistent.oaistatic.com\",
  \"https://*.oaistatic.com\",
  \"https://unpkg.com\",
  \"https://cdn.jsdelivr.net\",
  \"https://cdnjs.cloudflare.com\",
  \"https://cdn.skypack.dev\"
].join(\" \");

c.header(\"Content-Security-Policy\", [
  \"default-src 'self'\",
  `script-src 'self' 'unsafe-inline' 'unsafe-eval' ${trustedCdns}`,
  \"worker-src 'self' blob:\",
  \"child-src 'self' blob:\",
  `style-src 'self' 'unsafe-inline' ${trustedCdns}`,
  \"img-src 'self' data: https: blob:\",
  \"media-src 'self' data: https: blob:\",
  `font-src 'self' data: ${trustedCdns}`,
  \"connect-src 'self' https: wss: ws:\",
  \"frame-ancestors 'self'\"
].join(\"; \"));
```

### Iframe Sandbox

Located in `client/src/components/chat-v2/openai-app-renderer.tsx:518-530`:

```typescript
<iframe
  ref={iframeRef}
  src={widgetUrl}
  sandbox=\"allow-scripts allow-same-origin allow-forms allow-popups allow-popups-to-escape-sandbox\"
  title={`OpenAI Component: ${toolCall.name}`}
  allow=\"web-share\"
/>
```

**Sandbox Permissions:**

- `allow-scripts`: Enable JavaScript execution
- `allow-same-origin`: Allow localStorage access (required for state)
- `allow-forms`: Support form submissions
- `allow-popups`: Enable external link navigation
- `allow-popups-to-escape-sandbox`: Allow popup windows to load normally

**Security Trade-offs:**

- `allow-same-origin` + `allow-scripts` = Full JavaScript capabilities
- Required for React Router and modern frameworks
- Mitigated by CSP headers and origin validation
- Widgets should be treated as semi-trusted code

## Complete Data Flow Example

Let's trace a complete interaction where a widget calls a tool:

```mermaid
sequenceDiagram
    participant User
    participant Chat as ChatTab
    participant Widget as Widget Iframe
    participant Renderer as OpenAIComponentRenderer
    participant Backend as Backend API
    participant MCP as MCP Server

    Note over User,MCP: Initial Tool Execution
    User->>Chat: Execute weather tool
    Chat->>Backend: POST tool execution request
    Backend->>MCP: Call weather tool
    MCP-->>Backend: Return tool result with ui:// resource
    Backend-->>Chat: Return tool result
    Chat->>Renderer: Render OpenAI component

    Note over Renderer,Backend: Widget Setup
    Renderer->>Backend: POST widget data to store
    Backend-->>Renderer: Return success
    Renderer->>Renderer: Set widget URL

    Note over Widget,Backend: Widget Loading
    Widget->>Backend: GET widget container (Stage 1)
    Backend-->>Widget: Return container HTML
    Widget->>Widget: Change URL to root path
    Widget->>Backend: Fetch widget content (Stage 2)
    Backend->>MCP: Read resource from MCP
    MCP-->>Backend: Return HTML content
    Backend->>Backend: Inject window.openai script
    Backend-->>Widget: Return enhanced HTML
    Widget->>Widget: Render UI

    Note over User,MCP: Interactive Tool Call from Widget
    User->>Widget: Click Refresh Weather button
    Widget->>Widget: Call window.openai.callTool
    Widget->>Renderer: Send postMessage with tool request
    Renderer->>Backend: POST tool execution request
    Backend->>MCP: Call weather tool with new params
    MCP-->>Backend: Return new weather data
    Backend-->>Renderer: Return tool result
    Renderer->>Widget: Send postMessage with result
    Widget->>Widget: Update UI with new data
    Widget-->>User: Display updated weather

    Note over User,Chat: Followup Message
    User->>Widget: Click Ask about temperature button
    Widget->>Widget: Call window.openai.sendFollowupTurn
    Widget->>Renderer: Send postMessage with followup
    Renderer->>Chat: Call onSendFollowup callback
    Chat->>Chat: Set input and send message
    Chat->>Backend: POST chat message
    Note over Chat,Backend: Normal chat flow continues
```

## Development Guide

### Testing OpenAI SDK Widgets Locally

1. **Create a test MCP server with OpenAI SDK support:**

```typescript
// server.ts
import { Server } from \"@modelcontextprotocol/sdk/server/index.js\";
import { StdioServerTransport } from \"@modelcontextprotocol/sdk/server/stdio.js\";

const server = new Server(
  { name: \"test-widget-server\", version: \"1.0.0\" },
  { capabilities: { tools: {}, resources: {} } }
);

// Define tool with OpenAI output template
server.setRequestHandler(ListToolsRequestSchema, async () => ({
  tools: [{
    name: \"hello_widget\",
    description: \"Test widget with custom UI\",
    inputSchema: {
      type: \"object\",
      properties: {
        name: { type: \"string\" }
      }
    },
    _meta: {
      \"openai/outputTemplate\": \"ui://hello/display\"
    }
  }]
}));

// Tool returns structured content
server.setRequestHandler(CallToolRequestSchema, async (request) => {
  if (request.params.name === \"hello_widget\") {
    return {
      content: [{
        type: \"resource\",
        resource: {
          uri: \"ui://hello/display\",
          mimeType: \"text/html\"
        }
      }],
      _meta: {
        structuredContent: {
          greeting: `Hello, ${request.params.arguments.name}!`,
          timestamp: new Date().toISOString()
        }
      }
    };
  }
});

// Serve widget HTML
server.setRequestHandler(ReadResourceRequestSchema, async (request) => {
  if (request.params.uri === \"ui://hello/display\") {
    return {
      contents: [{
        uri: \"ui://hello/display\",
        mimeType: \"text/html\",
        text: `
<!DOCTYPE html>
<html>
<head>
  <meta charset=\"UTF-8\">
  <title>Hello Widget</title>
  <style>
    body { font-family: system-ui; padding: 20px; }
    button { padding: 10px 20px; margin: 10px 0; }
  </style>
</head>
<body>
  <div id=\"app\"></div>
  <script>
    // Access tool input/output
    const input = window.openai.toolInput;
    const output = window.openai.toolOutput;

    // Render UI
    document.getElementById('app').innerHTML = \`
      <h1>\${output.greeting}</h1>
      <p>Time: \${output.timestamp}</p>
      <button id=\"refresh\">Refresh</button>
      <button id=\"followup\">Send Followup</button>
    \`;

    // Call tool on button click
    document.getElementById('refresh').addEventListener('click', async () => {
      const result = await window.openai.callTool('hello_widget', {
        name: input.name + ' (refreshed)'
      });
      console.log('Tool result:', result);
    });

    // Send followup message
    document.getElementById('followup').addEventListener('click', () => {
      window.openai.sendFollowupTurn('Tell me more about greetings');
    });
  </script>
</body>
</html>
        `
      }]
    };
  }
});

const transport = new StdioServerTransport();
server.connect(transport);
```

2. **Add server to MCPJam Inspector config:**

```json
{
  \"mcpServers\": {
    \"test-widget\": {
      \"command\": \"node\",
      \"args\": [\"path/to/server.ts\"]
    }
  }
}
```

3. **Test in Inspector:**
   - Connect to server in Servers tab
   - Navigate to Chat tab
   - Execute: "Call the hello_widget tool with name John"
   - Widget should render with interactive buttons

### Debugging Widget Issues

**Common Problems:**

1. **Widget doesn't load (404)**
   - Check that `widgetDataStore` contains toolId
   - Verify storage TTL hasn't expired (1 hour default)
   - Confirm MCP server returns valid HTML for `ui://` resource

2. **`window.openai` is undefined**
   - Verify script injection in Stage 2 content endpoint
   - Check browser console for CSP violations
   - Ensure `<head>` tag exists in HTML for injection

3. **Widget data not found (404)**
   - Check that widget data was successfully stored via POST `/api/mcp/openai/widget/store`
   - Verify toolId matches between store and load requests
   - Check server logs for storage errors

4. **Tool calls timeout**
   - Check network tab for `/api/mcp/tools/execute` failures
   - Verify MCP server is connected and responsive
   - Increase timeout in `callTool` implementation (default: 30s)

5. **React Router 404 errors**
   - Confirm Stage 1 executes `history.replaceState('/')` before loading
   - Check that widget uses `BrowserRouter` not `HashRouter`
   - Verify `<base href=\"/\">` is present in HTML

6. **State doesn't persist**
   - Check localStorage in browser DevTools
   - Verify `widgetStateKey` format: `openai-widget-state:${toolName}:${toolId}`
   - Confirm `setWidgetState` postMessage handler is working

7. **Theme not updating**
   - Check that `openai:set_globals` messages are being sent from parent
   - Verify widget has event listener for theme changes
   - Inspect `window.openai.theme` value in widget console

8. **Modal not opening**
   - Verify `requestModal` postMessage is being sent
   - Check that Dialog component state is updating (`modalOpen`)
   - Inspect modal iframe URL includes `view_mode=modal` query param
   - Confirm `window.openai.view.mode` is `'modal'` in modal iframe

9. **State not syncing between inline and modal**
   - Check that both iframes are mounted and have valid contentWindow
   - Verify `openai:pushWidgetState` messages are being sent
   - Inspect localStorage for widget state key
   - Confirm both views have `openai:widget_state` event listener

**Debug Tools:**

```javascript
// Add to widget for debugging
window.addEventListener("message", (e) => {
  console.log("[Widget] Received message:", e.data);
});

// Monitor all postMessage calls
const originalPostMessage = window.parent.postMessage;
window.parent.postMessage = function (...args) {
  console.log("[Widget] Sending message:", args[0]);
  return originalPostMessage.apply(window.parent, args);
};

// Check openai API availability
console.log("openai API:", window.openai);
console.log("toolInput:", window.openai?.toolInput);
console.log("toolOutput:", window.openai?.toolOutput);
```

### Extending the Implementation

**Adding New OpenAI API Methods:**

1. Update server-side injection script (`server/routes/mcp/resources.ts:250-376`)
2. Add postMessage handler in parent (`client/src/components/chat/openai-component-renderer.tsx:118-196`)
3. Update TypeScript types if needed

**Example: Adding a custom method:**

```typescript
// 1. Server-side API injection (server/routes/mcp/openai.ts)
openaiAPI.customMethod = async function(options) {
  window.parent.postMessage({
    type: 'openai:customMethod',
    data: options
  }, '*');
};

// 2. Parent-side handler (client/src/components/chat-v2/openai-app-renderer.tsx)
case \"openai:customMethod\":
  handleCustomMethod(event.data.data);
  break;
```

## Performance Considerations

### Widget Data Storage

- **TTL**: 1 hour default, configurable in `resources.ts:22`
- **Cleanup**: Runs every 5 minutes
- **Memory**: Each widget stores ~1-10KB (toolInput + toolOutput)
- **Scale**: 1000 concurrent widgets ≈ 10MB memory
- **Recommendation**: For production, use Redis instead of Map

### Iframe Rendering

- **Initial Load**: 200-500ms (Stage 1 + Stage 2 + resource fetch)
- **Tool Calls**: 100-300ms (postMessage + backend + MCP)
- **Optimization**:
  - Cache MCP resource reads (currently disabled with `no-cache`)
  - Preload widget data before iframe creation
  - Use service workers for offline support

### postMessage Overhead

- **Latency**: 5-15ms per message round-trip
- **Payload**: JSON serialization for all data
- **Bottleneck**: Large tool results (>1MB) slow down significantly
- **Mitigation**: Use streaming or chunked responses for large data

## Security Best Practices

1. **Validate postMessage Origins:**

   ```typescript
   if (event.source !== iframeRef.current?.contentWindow) return;
   ```

2. **Sanitize Tool Parameters:**

   ```typescript
   const params = JSON.parse(JSON.stringify(event.data.params)); // Deep clone
   // Validate against tool schema before execution
   ```

3. **Limit Widget Capabilities:**
   - Only expose necessary MCP tools to widgets
   - Implement rate limiting on tool calls
   - Restrict network access via CSP

4. **Content Security Policy:**
   - Remove `unsafe-eval` if possible (breaks some frameworks)
   - Whitelist only trusted CDNs
   - Consider using nonces for inline scripts

5. **Audit Widget Code:**
   - Widgets have semi-trusted status
   - Review HTML content from MCP servers
   - Scan for XSS vulnerabilities
   - Monitor for suspicious postMessage patterns

## Related Files

- `client/src/components/tools/ResultsPanel.tsx` - Detects OpenAI components
- `client/src/components/chat-v2/openai-app-renderer.tsx` - Renders iframes, handles widget lifecycle, and manages display modes
- `client/src/components/chat-v2/thread.tsx` - Manages PiP state across all widgets in the thread
- `client/src/components/ChatTabV2.tsx` - Chat integration with transform isolation for z-index stacking
- `server/routes/mcp/openai.ts` - Widget storage, serving, and OpenAI bridge injection
- `server/routes/mcp/index.ts` - Mounts OpenAI routes at `/openai`
- `client/src/lib/mcp-tools-api.ts` - Tool execution API

## Resources

- [OpenAI Apps SDK - Custom UX Guide](https://developers.openai.com/apps-sdk/build/custom-ux)
- [OpenAI Apps SDK - API Reference](https://developers.openai.com/apps-sdk/reference)
- [MCP Specification](https://modelcontextprotocol.io/)
- [MCPJam Inspector Repository](https://github.com/MCPJam/inspector)

## Contributing

When contributing to the OpenAI SDK integration:

1. **Test with real MCP servers** - Don't just mock the API
2. **Check security implications** - All changes to iframe/postMessage code need review
3. **Update this documentation** - Keep architecture diagrams current
4. **Add debug logging** - Use `console.log` with `[OpenAI Widget]` prefix
5. **Consider backwards compatibility** - Existing widgets should continue working

For questions or issues, open a GitHub issue or join our [Discord community](https://discord.gg/JEnDtz8X6z).
